Background replacement from video images captured by a plenoptic camera

ABSTRACT

Embodiments disclosed herein provide systems, methods, and computer readable media for replacing a video background in real-time. The video comprises a plurality of image frames. In a particular embodiment, a method provides generating a range image from a subject image frame of the plurality of image frames. The range image indicates pixel distances from a plenoptic camera that captured the plurality of image frames. The method further provides identifying background pixels that represent a background portion of the subject image frame based on the range image and replacing the background pixels with replacement background pixels in the subject image frame.

RELATED APPLICATIONS

This application is related to and claims priority to U.S. patentapplication Ser. No. 14/921,652, titled “BACKGROUND REPLACEMENT FROMVIDEO IMAGES CAPTURED BY A PLENOPTIC CAMERA,” filed Oct. 23, 2015, whichis related to and claims priority to U.S. Provisional Patent ApplicationNo. 62/137,921, entitled “REAL-TIME BACKGROUND REPLACEMENT IN AGENTVIDEO CALLS FOR CUSTOMER CARE,” filed on Mar. 25, 2015, and which areboth hereby incorporated by reference in their entirety.

TECHNICAL FIELD

Aspects of the disclosure are related to video image manipulation and,in particular, to replacing background portions of a video image.

TECHNICAL BACKGROUND

Communication and computing systems allow users to exchange videocommunication from almost anywhere that a wired or wireless networkconnection is available. User devices including personal workstations,laptops, smartphones, tablets, and the like, all have the ability toexchange video communications. For example, users can make personalvideo calls from their cellphones or an enterprise's contact center candistribute agents throughout the world. In the latter example, contactcenter agents may even be able to work from home rather than a contactcenter facility and still participate in video communications withcallers.

Unlike audio or text communications, video communications capture boththe user and, in the background, portions of the location in which theuser is located. In some cases, that location may not be appropriate forthe video communication or otherwise is less preferred than anotherbackground. For example, a user may need to make a business relatedvideo call while on a beach vacation. It would likely not be appropriatefor the beach to be in the background while the user is participating inthat video call. In an alternative example, a contact center that allowsits agents to work from home may still want a consistent backgroundbehind all agents regardless of their location. However, currenttechniques for background replacement require a relatively large amountof processing resources to identify backgrounds, sometimes requiringsolid color backgrounds (e.g. blue/green screen used in chroma keying),or require hardware in addition to the camera capturing video, such asan infrared blaster and sensor.

Overview

Embodiments disclosed herein provide systems, methods, and computerreadable media for replacing a video background in real-time. The videocomprises a plurality of image frames. In a particular embodiment, amethod provides generating a range image from a subject image frame ofthe plurality of image frames. The range image indicates pixel distancesfrom a plenoptic camera that captured the plurality of image frames. Themethod further provides identifying background pixels that represent abackground portion of the subject image frame based on the range imageand replacing the background pixels with replacement background pixelsin the subject image frame.

In some embodiments, generating the range image comprises calculating apixel distance for each pixel in at least a subset of pixels in thesubject image frame based on perspectives of each pixel as captured bythe plenoptic camera.

In some embodiments, the replacement background pixels represent animage frame of a second video.

In some embodiments, identifying the background pixels comprisesidentifying pixels of the subject image frame having pixel distancesgreater than a threshold distance from the plenoptic camera.

In some embodiments, the threshold distance comprises a pixel distancegreater than that of pixels in the subject image frame representing auser in the subject image frame.

In some embodiments, the method further provides establishing a videocommunication between a first communication system having the plenopticcamera and a second communication system.

In some embodiments, after replacing the background pixels, the methodprovides transferring the subject image frame to the secondcommunication system.

In some embodiments, the first communication system comprises one of aplurality of communication systems for a contact center.

In some embodiments, the generating the range image, identifying thebackground pixels, and replacing the background pixels are eachperformed by the first communication device.

In some embodiments, the method further provides generating a secondrange image from a second subject image frame of the plurality of imageframes. The second range image indicates second pixel distances from theplenoptic camera. The method also provides identifying second backgroundpixels that represent a second background portion of the second subjectimage frame based on the second range image and replacing the secondbackground pixels with second replacement background pixels in thesubject image frame.

In another embodiment, a video processing system is provided includingan interface and a processing system. The interface is configured toreceive the video from a plenoptic camera that captured the plurality ofimage frames. The processing system is configured to generate a rangeimage from a subject image frame of the plurality of image frames,identify background pixels that represent a background portion of thesubject image frame based on the range image, and replace the backgroundpixels with replacement background pixels in the subject image frame.

BRIEF DESCRIPTION OF THE DRAWINGS

Many aspects of the disclosure can be better understood with referenceto the following drawings. While several implementations are describedin connection with these drawings, the disclosure is not limited to theimplementations disclosed herein. On the contrary, the intent is tocover all alternatives, modifications, and equivalents.

FIG. 1 illustrates a video capture environment for replacing a videoimage background.

FIG. 2 illustrates a method of operating the video capture environmentto replace a video image background.

FIG. 3 illustrates a video communication environment for replacing avideo image background.

FIG. 4 illustrates a method of operating the video communicationenvironment to replace a video image background.

FIG. 5 illustrates an operational scenario of the video communicationenvironment to replace a video image background.

FIG. 6 illustrates an operational scenario of the video communicationenvironment to replace a video image background.

FIG. 7 illustrates video images from an operational scenario of thevideo communication environment to replace a video image background.

FIG. 8 illustrates a video processing system for replacing a video imagebackground.

TECHNICAL DISCLOSURE

Plenoptic cameras (also called light field cameras) are able to capturelight intensity of a scene and information about direction the capturedlight was travelling. One way of implementing a plenoptic camera is toplace an array of micro lenses between a camera's image sensor and itsprimary lens. There may be as few as two micro lenses up to any numberdepending on the size of the lens and the size of the image sensor. Thismicro lens arrangement allows each of the micro lenses refocus the lightcoming through the primary lens onto a portion of the image sensor undereach respective micro lens. Accordingly, due to their close proximity,adjacent micro lenses focus similar images onto their correspondingsensor portions. However, since each micro lens is in a slightlydifferent position from one another, the light for an image that reachesone micro lens' portion of the image sensor is travelling in a differentdirection than the light that reaches another micro lens' portion of theimage sensor. The perspective of the image focused onto one portion ofthe sensor is therefore slightly different than the image on anotheradjacent portion of the sensor. The difference in perspective is greaterfor micro lenses located further apart. It is from this difference inperspective that a relative distance can be calculated for each pixel inan image.

FIG. 1 illustrates video capture environment 100 in an example scenariofor replacing a video image background. Video capture environment 100includes video processing system 101 and plenoptic camera 102. Videoprocessing system 101 and plenoptic camera 102 communicate overcommunication link 111.

While shown separately, plenoptic camera 102 and video processing system101 may be combined into a single device, such as a smartphone, tablet,personal computer, video display monitor, or other type of computingdevice. Plenoptic camera 102 captures video image frames that togetherin sequence form a video image. Since each image frame captured byplenoptic camera 102 includes multiple images captured by differentportions of an image sensor within plenoptic camera 102, videoprocessing system 101 selects between the multiple images, and in somecases combines at least a portion of the multiple images, in order tocreate an image. It is these post processed images that are combined tocreate a video that may be played back to a user, transferred to anothersystem, stored in a storage system, or used for some other purpose.

FIG. 2 illustrates a method 200 for operating video capture environment100 to replace a video image background. Method 200 provides videoprocessing system 101 generating a range image from subject image frame121 of image frames captured by plenoptic camera 102 (step 201). Therange image indicates pixel distances from plenoptic camera 102.Specifically, the range image indicates pixel distances relative toplenoptic camera 102's position. The pixel distances may correspond toeach individual pixel or to a group of pixels. In some cases, only pixeldistances for pixels that have changed since a prior image frame wascaptured may be determined to avoid using processing resources todetermine already known pixel distances. It should be understood thateach of the pixel distances correspond to pixels of subject image frame121 after processing the multiple images captured by plenoptic camera102's sensor into a single image that can be displayed to a user, asdescribed above. However, as also described above, it is the multipleperspectives offered by the multiple images that are used to determinethe relative distances from plenoptic camera 102 of each pixel in thepost-processed subject image frame 121. The pixel distances themselvesmay be derived using any known method of determining pixel distancescaptured by a plenoptic camera.

Method 200 then provides video processing system 101 identifyingbackground pixels that represent a background portion of the subjectimage frame based on the range image (step 202). A pixel distancethreshold may be used to determine whether a pixel distance correspondsto a background pixel. The threshold distance may be a constant distanceor may be adaptable depending on the scene captured in subject imageframe 121. For example, image processing may be performed on subjectimage frame 121 or on a previously captured image frame in order toidentify a user in the frame. The pixels representing the user insubject image frame 121 may then be identified as the pixels havingroughly the same pixel distance and location in subject image frame 121.The threshold distance may then be set to a value that indicates allpixels further from plenoptic camera 102 than those representing theuser in subject image frame 121 correspond to background pixels.Preferably the threshold distance is set great enough so as to accountfor normal variations in a pixel distance for the user that is cause bythe user's movement during the capture of the video.

In some examples, certain items that may otherwise be considered in thebackground of subject image frame 121 (e.g. a plant, window, etc.) maybe identified in a similar manner as described for the user above andmay be differentiated from the rest of the background based on one ormore additional threshold distances. Likewise, items having pixels withdistances on either side of the pixel distance threshold may not beconsidered part of the background. For example, an ell shaped desk whereone portion is in front of a user and another portion that extends tothe side of the user may extend beyond the threshold pixel distance.However, the desk as a whole may be excluded from the background pixels.

Once the background pixels are identified, method 200 provides videoprocessing system 101 replacing the background pixels with replacementbackground pixels in subject image frame 321 (step 203). The replacementbackground pixels may comprise pixels of any color or may be multiplecolors. As a whole the pixels may represent a still image or mayrepresent pixels in a frame of a background video, whereby thereplacement pixels in subsequent subject image frames may change inaccordance with the background video. In some cases, the imagerepresented by the replacement background pixels may be resized orotherwise changed in order to best fit in the space of subject imageframe 121 that was occupied by the background pixels. Similarly,portions of the image represented by the replacement background pixelsmay not be included in the replacement pixels if the background pixelsbeing replaced do not allow for the entire background image (i.e. thenon-background pixels of the image remain where those non-includedportions would be placed). For example, a user in the foreground ofsubject image frame 121 may displace a corresponding portion of thebackground image just as the user would be obscuring whatever was behindthe user in the original background. Moreover, in some cases, thereplacement background pixels may be implemented in such a way as tomaintain the look in subject image frame 121 of the background beingbehind the image portions that remain in the foreground. For example, abackground image may be presented by the replacement pixels such that aviewer of subject image frame 121 would see it as being a distance (e.g.a number of feet) behind any object in the foreground.

It should be understood that while method 200 regarding subject imageframe 121 works to replace the background of a still image, method 200is repeated on the other image frames captured by plenoptic camera 102in order to remove the background from the video. Advantageously,replacing the background using method 200 should be less resourceintensive than other background detection and replacement methods, whichmakes method 200 suitable for real-time background replacement duringlive video transmissions or real-time video communications. For example,method 200 does not use the image processing techniques required todetect an image background based on the visuals of the image itself.Likewise, method 200 is not affected by cluttered background colors in acaptured image or motion in the background between frames, whichtherefore removes the advantage of solid colored backgrounds (e.g.blue/green screens for chroma keying). Moreover, additional sensors usedto gather a depth of a scene, such as infrared sensors, are not needed.

Referring back to FIG. 1, video processing system 101 comprises acomputer processor system and an interface for communicating withplenoptic camera 102. Video processing system 101 may also include othercomponents such as a router, server, data storage system, userinterface, and power supply. Video processing system 101 may reside in asingle device or may be distributed across multiple devices. Videoprocessing system 101 may be a video communication server, conferencingsystem, application server, personal computer workstation, smartphone,tablet, or some other computing system—including combinations thereof.

Plenoptic camera 102 comprises a lens, a digital image sensor, and aninterface for communicating with video processing system 101. Plenopticcamera 102 may further include a power supply, user interface, or someother component for operating a plenoptic camera. While shownseparately, plenoptic camera 102 may be incorporated into videoprocessing system 101. Plenoptic camera 102 may be a standard plenopticcamera or a focused plenoptic camera, which both use additional microlenses in addition to the lens above, may be a coded aperture camera, ormay be some other type of light field camera.

Communication link 111 use metal, glass, air, space, or some othermaterial as the transport media. Communication link 111 could be aninternal link, such as a system bus, if plenoptic camera 102 and videoprocessing system 101 are part of a single device. Communication link111 could use various communication protocols, such as Time DivisionMultiplex (TDM), Internet Protocol (IP), Ethernet, communicationsignaling, Code Division Multiple Access (CDMA), Evolution Data Only(EVDO), Worldwide Interoperability for Microwave Access (WIMAX), GlobalSystem for Mobile Communication (GSM), Long Term Evolution (LTE),Wireless Fidelity (WIFI), High Speed Packet Access (HSPA), or some othercommunication format—including combinations thereof. Communication link111 could be direct links or may include intermediate networks, systems,or devices.

FIG. 3 illustrates video communication environment 300 in an examplescenario for replacing a video image background. Video communicationenvironment 300 includes user system 301, user system 302, videocommunication system 303, and communication network 304. User system 302includes plenoptic camera 331. User system 301 and communication network304 communicate over communication link 311. User system 302 andcommunication network 304 communicate over communication link 312. Videocommunication system 303 and communication network 304 communicate overcommunication link 313.

Communication network 304 comprises network elements that providecommunications services. Communication network 304 may compriseswitches, wireless access nodes, Internet routers, network gateways,application servers, computer systems, communication links, or someother type of communication equipment—including combinations thereof.Communication network 304 may be a single network (e.g. local areanetwork, wide area network, the Internet, etc.) or may be a collectionof networks.

Video communication system 303 is a system that facilitates the exchangeof real-time video communications between user systems 301 and 302. Thevideo communications may be unidirectional from user system 301 to usersystem 302 or may be bidirectional should user system 302 be equippedwith a camera or other component necessary for video communications.Video communication system 303 may be a video conferencing system thatmanages video conferences between multiple parties. Thus, while videocommunication environment 300 only includes user systems 301 and 302,video communication system 303 may be able to manage conferences betweenmore than two user systems. Similarly, video communication system 303may be used to handle video calls with a contact center for an entityallowing users (e.g. customers of the entity) to communicate with agentsusing video. Also, it should be understood that some embodiments mayremove the need for video communication system 303 altogether, as may bethe case in a peer-to-peer video communication between user systems 301and 302.

FIG. 4 illustrates a method 400 for operating video communicationenvironment 300 to replace a video image background. Method 400 providesthat a video call is established between user system 301 and user system302 (step 401). The video call may be a one-on-one video call betweenuser systems 301 and 302, may be a conference call for which usersystems 301 and 302 are a part, or may be some other type of videocommunication. The video call may be initiated from either of usersystems 301 and 302, by video communication system 303, or may beinitiated from some other system. The video call may only transfer videofrom user system 301 to user system 302, although, user system 302 maytransfer video to user system 301 if properly equipped. In a one-wayvideo situation, audio and video may be transferred from user system 301to user system 302 while only audio is transferred from user system 302to user system 301.

During the video call, video is captured of scene 321, which may includea user of user system 301, and pixel distances are determined for theimage frames that make up the video (step 402). As in method 200described above, the pixel distances of each image frame can bedetermined from the difference(s) in perspective captured by differentportions of plenoptic camera 331's image sensor. Pixel distances thatexceed a threshold distance are then identified to identifycorresponding pixels as being in the background of the video (step 403).The background pixels of each image frame are then replaced withreplacement background pixels in order to replace the background of thevideo (step 404).

After replacing the background, the image frames are transferred to usersystem 302 as video (step 405). It should be understood that each frameof the video is captured and processed to replace the background insubstantially real-time so as not to adversely effect the userexperience of the video call. Accordingly, the video is received by usersystem 302 without noticeable delay caused by the background imagereplacement.

FIG. 5 illustrates an operational scenario 500 of video communicationenvironment 300 to replace a video image background. In scenario 500, avideo communication session is established between user systems 301 and302. While video communication system 303 may facilitate theestablishment of the video communication, media for the videocommunication session (e.g. audio, video, etc.) does not necessarilypass through video communication system 303. Scene 321 for the videocommunication is captured by plenoptic camera 331, which passes thevideo of scene 321 to user system 301 for processing. During processing,user system 301 generates a video image suitable for being presented toa user. That is, user system 301 creates a video image from the multipleimages captured by plenoptic camera 331. Additionally, user system 301identifies and replaces the background of that video image using pixeldistance information from the multiple images captured by plenopticcamera 331, as described above. The video is then transferred to usersystem 302 for display to a user of user system 302.

FIG. 6 illustrates an operational scenario 600 of video communicationenvironment 300 to replace a video image background. As in scenario 500,in scenario 600 video communication system 303 establishes a videocommunication between user systems 301 and 302. However, in scenario 600at least the video from user system 301 is transferred through videocommunication system 303 from user system 301 before reaching usersystem 302. Accordingly, after scene 321 is captured by plenoptic camera331 and passed to user system 301, the video is transferred by usersystem 301 to video communication system 303. The video as transferredto video communication system 303 includes the multiple images capturedby plenoptic camera 331 and has yet to be processed into a video imagefor display. As such, video communication system 303 is able to and doesperform the background identification and replacement for the videoinstead of user system 301 as was the case in scenario 500. Inalternative examples, user system 301 may generate a range image andtransfer the range image along with a displayable video image to videocommunication system 303 rather than transferring the multiple images ascaptured by plenoptic camera 331.

The video having the replaced background is then transferred from videocommunication system 303 to user system 302 where the video can bepresented to a user. In some examples, video communication system 303may also transfer the video to user system 301 for presentation to auser of user system 301. This allows the user of user system 301 to viewthe video after the background has been replaced even though user system301 did not itself process the video, as would be the case in scenario500.

FIG. 7 illustrates video images from an operational scenario 700 ofvideo communication environment 300 to replace a video image background.At step 1, video is captured of scene 321 via plenoptic camera 331.Subject image frame 701 is an image frame of the captured video. Inparticular, subject image frame 701 comprises a frame of the video afterprocessing the multiple images captured by plenoptic camera 331 into asingle image that can be presented to a user. Subject image frame 701shows user 711 sitting at desk 714 with a shelf 713 and a window 712located on a wall behind user 711. It should be understood that whilescenario 700 illustrates subject image frame 701, subject image frame701 may never be displayed to a user but, rather, may merely exist asdata in video communication environment 300.

In this example, user 711 is an agent working for a bank called ExampleBank. The captured video is part of a video call with a customer ofExample Bank. User 711 is one of many agents employed for customerservice by Example Bank and Example Bank would like each agent to have aconsistent background when talking with a customer over a videocommunication.

Therefore, at step 2, a range image 702 of subject image frame 701 isgenerated. As with subject image frame 701, range image 702 may neveractually be displayed but, instead, may merely exist as data in videocommunication environment 300. Range image 702 illustrates pixeldistances of the various elements within subject image frame 701. Atdistance level 721 is the wall, shelf 713 is shown as at distance level722, user 711 is at level 723, and desk 714 is at distance level 724.For clarity, in this example, range image 702 has only 4 distance levelsindicating pixel distances however any granularity of distances,including infinite granularity, may be used. For example, the pixeldistances for pixels representing desk 714 may get progressively largertowards user 711 and may also be able to distinguish the keyboardthereon.

At step 3, background pixels are identified and replaced with a newbackground image to produce image frame 703. Specifically, it isdetermined that all pixels having pixel distances greater than distancelevel 723 are behind user 711 and are background. Pixels in distancelevels 721 and 722 are therefore identified as background pixels thatwill be replaced. In this example, the background pixels are replacedwith pixels showing a banner for Example Bank with Example Bank's sloganthereunder on a plain background. Since user 711 is not part of thebackground, user 711 “obscures” a portion of the slogan. However, insome examples, the banner may be resized or otherwise changed so thatthe entirety of the slogan can be seen without being blocked by user711.

The above steps are repeated on all frames of the video so that theExample Bank banner is displayed as the background of the video as awhole. To account for movement of user 711 or other objects captured inthe foreground of the video, the pixels representing the background ofthe captured image may change over time. For example, in later frames ofthe video, user 711 may move to their left and obscure the “your” in theslogan while allowing the “serve all” portion (currently obscured inimage frame 703) to be seen.

FIG. 8 illustrates video processing system 800. Video processing system800 is an example of video processing system 101, although system 101could use alternative configurations. Video processing system 800comprises communication interface 801, user interface 802, andprocessing system 803. Processing system 803 is linked to communicationinterface 801 and user interface 802. Processing system 803 includesprocessing circuitry 805 and memory device 806 that stores operatingsoftware 807. Video processing system 800 may include other well-knowncomponents such as a battery and enclosure that are not shown forclarity. Video processing system 800 may be a personal computer, tablet,application server, conferencing system, private branch exchange, orsome other computing apparatus—including combinations thereof.

Communication interface 801 comprises components that communicate overcommunication links, such as network cards, ports, RF transceivers,processing circuitry and software, or some other communication devices.Communication interface 801 may be configured to communicate overmetallic, wireless, or optical links. Communication interface 801 may beconfigured to use TDM, IP, Ethernet, optical networking, wirelessprotocols, communication signaling, or some other communicationformat—including combinations thereof.

User interface 802 comprises components that interact with a user toreceive user inputs and to present media and/or information. Userinterface 802 may include a speaker, microphone, camera, buttons,lights, display screen, touch screen, touch pad, scroll wheel,communication port, or some other user input/output apparatus—includingcombinations thereof. In some examples, user interface 802 includes aplenoptic camera. User interface 802 may be omitted in some examples.

Processing circuitry 805 comprises microprocessor and other circuitrythat retrieves and executes operating software 807 from memory device806. Memory device 806 comprises a non-transitory storage medium, suchas a disk drive, flash drive, data storage circuitry, or some othermemory apparatus. Processing circuitry 805 is typically mounted on acircuit board that may also hold memory device 806 and portions ofcommunication interface 801 and user interface 802. Operating software807 comprises computer programs, firmware, or some other form ofmachine-readable processing instructions. Operating software 807includes range image module 808 and background replacement module 809.Operating software 807 may further include an operating system,utilities, drivers, network interfaces, applications, or some other typeof software. When executed by processing circuitry 805, operatingsoftware 807 directs processing system 803 to operate video processingsystem 800 as described herein.

In particular, range image module 808 directs processing system 803 togenerate a range image from a subject image frame of a plurality ofimage frames included in a video. The range image indicates pixeldistances from a plenoptic camera that captured the plurality of imageframes. Background replacement module 809 directs processing system 803to identify background pixels that represent a background portion of thesubject image frame based on the range image. Background replacementmodule 809 also directs processing system 803 to replace the backgroundpixels with replacement background pixels in the subject image frame.

The included descriptions and figures depict specific implementations toteach those skilled in the art how to make and use the best option. Forthe purpose of teaching inventive principles, some conventional aspectshave been simplified or omitted. Those skilled in the art willappreciate variations from these implementations that fall within thescope of the invention. Those skilled in the art will also appreciatethat the features described above can be combined in various ways toform multiple implementations. As a result, the invention is not limitedto the specific implementations described above, but only by the claimsand their equivalents.

What is claimed is:
 1. A method of replacing a video background inreal-time, wherein the video comprises a plurality of image frames, themethod comprising: receiving the video captured by a plenoptic camera ofa first user system; generating a range image from a subject image frameof the plurality of image frames, wherein the range image indicatespixel distances from the plenoptic camera; identifying background pixelsthat represent a background portion of the subject image frame based onthe range image; replacing the background pixels with replacementbackground pixels in the subject image frame; and after replacing thebackground pixels, transmitting the subject image frame to a second usersystem over a video communication session.
 2. The method of claim 1,wherein receiving the video comprises: receiving the video from thefirst user system.
 3. The method of claim 1, wherein receiving the videocomprises: receiving the video from the plenoptic camera.
 4. The methodof claim 3, wherein transmitting the subject image frame comprises:transmitting the video from the first user system to the second usersystem.
 5. The method of claim 1, wherein the first user system isassociated with a contact center and the replacement background pixelsrepresent an image associated with the contact center.
 6. The method ofclaim 1, wherein the subject image frame includes a user of the firstuser system.
 7. The method of claim 1, comprising: generating a secondrange image from a second subject image frame of the plurality of imageframes, wherein the second range image indicates second pixel distancesfrom the plenoptic camera; identifying second background pixels thatrepresent a second background portion of the second subject image framebased on the second range image; replacing the second background pixelswith the replacement background pixels in the second subject imageframe; and after replacing the second background pixels, transmittingthe second subject image frame to the second user system over the videocommunication session.
 8. A method of replacing a video background,wherein the video comprises a plurality of image frames, the methodcomprising: receiving the video captured by a plenoptic camera;generating a range image from a subject image frame of the plurality ofimage frames, wherein the range image indicates pixel distances from theplenoptic camera; identifying background pixels that represent abackground portion of the subject image frame based on the range image;replacing the background pixels with replacement background pixels inthe subject image frame; and after replacing the background pixels,storing the video in a storage system.
 9. The method of claim 8,comprising: playing the video back to a user from the storage system.10. The method of claim 8, comprising: transmitting the video to a usersystem.
 11. The method of claim 8, wherein receiving the videocomprises: receiving the video from a first user system.
 12. The methodof claim 8, wherein receiving the video comprises: receiving the videofrom the plenoptic camera.
 13. The method of claim 8, wherein an objectis represented by foreground pixels of the subject image frame.
 14. Amethod of replacing a video background, wherein the video comprises aplurality of image frames, the method comprising: generating a rangeimage from a subject image frame of the plurality of image frames,wherein the range image indicates pixel distances from a plenopticcamera that captured the plurality of image frames; and replacingbackground pixels in the subject image frame with replacement backgroundpixels based on the range image.
 15. The method of claim 14, comprising:after replacing the background pixels, storing the video in a storagesystem.
 16. The method of claim 15, comprising: playing the video backto a user from the storage system.
 17. The method of claim 14,comprising: after replacing the background pixels, transmitting thevideo to a second user system over a video communication session. 18.The method of claim 14, comprising: receiving the video from a firstuser system.
 19. The method of claim 14, comprising: receiving the videofrom the plenoptic camera.
 20. The method of claim 14, comprising:generating a second range image from a second subject image frame of theplurality of image frames, wherein the second range image indicatessecond pixel distances from the plenoptic camera; and replacing thesecond background pixels with the replacement background pixels in thesecond subject image frame based on the second range image.